I may be wrong here, but it was my understanding that there is nothing pagan in origin about Father's Day. It's a relatively modern invention which was designed to counterbalance Mother's Day. And no doubt to make more money for retailers.
So, in theory you should be able to mark Father's Day even if you're PIMO or PIMI.